{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_qNSzzyaCbD"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "jmjh290raIky"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# <div dir=\"rtl\"> الترجمة الآلية باستخدام الشبكات العصبية مع آلية الانتباه </div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOpGoE2T-YXS"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"right\" dir=\"rtl\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/nmt_with_attention?hl=ar\">\n",
        "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
        "    إفتح المحتوى على موقع TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ar/tutorials/text/nmt_with_attention.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    تفاعل مع المحتوى على Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ar/tutorials/text/nmt_with_attention.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    اعرض المصدر على Github</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ar/tutorials/text/nmt_with_attention.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />نزّل الدّفتر</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AfOT1Ofy9JXi"
      },
      "source": [
        "<div dir=\"rtl\">\n",
        "\n",
        "Note: قامت مجموعة المتطوعين في مشروع Tensorflow بترجمة هذا المحتوى. نظرًا لأن هذه الترجمات تعتمد على قاعدة أقصى الجهد (best-effort) ، فلا نضمن أنها انعكاس دقيق وحديث [للمحتوى الرسمي باللغة الإنجليزية](https://www.tensorflow.org/?hl=en). إذا كانت لديك اقتراحات لتحسين هذه الترجمة ، يرجى إرسال \"Pull request\" إلى مشروع [tensorflow/docs-l10n](https://github.com/tensorflow/docs-l10n). للتطوع أو مراجعة ترجمات المجموعة يرجى ارسال ايميل إلى docs@tensorflow.org.\n",
        "\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CiwtNgENbx2g"
      },
      "source": [
        "<div dir=\"rtl\">\n",
        "\n",
        " نقوم في هذا الدفتر بتعلّم كيفية تدريب نموذج تسلسل إلى تسلسل (seq2seq) للترجمة من الإسبانية إلى الإنجليزية. هذا مثال متقدم يفترض بعض المعرفة بنماذج التسلسل إلى تسلسل. بعد تدريب النموذج في هذا الدفتر، ستتمكن من إدخال جملة إسبانية ، مثل: \n",
        "\n",
        "</div> \n",
        "\n",
        "\"*¿todavia estan en casa؟*\"  \n",
        "\n",
        "\n",
        "<div dir=\"rtl\">  واستخراج الترجمة الإنجليزية:\n",
        "</div>  \n",
        "\n",
        "\"*are you still at home?*\"\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "جودة الترجمة مقبولة لمثال بسيط ، ولكن ربما يكون الرسم البياني للانتباه (attention) الناتج أكثر إثارة للاهتمام. فهو يوضح  أجزاء الجملة المدخلة التي لفتت انتباه النموذج أثناء الترجمة:\n",
        "\n",
        "</div>\n",
        "\n",
        "<img src=\"https://tensorflow.org/images/spanish-english.png\" alt=\"spanish-english attention plot\">\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "Note: \n",
        "يستغرق هذا المثال حوالي 10 دقائق للتشغيل على GPU P100.\n",
        "\n",
        "</div> "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tnxXKDjq3jEL"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import unicodedata\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import io\n",
        "import time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfodePkj3jEa"
      },
      "source": [
        "## <div dir=\"rtl\"> تنزيل وتحضير مجموعة البيانات </div>\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "سنستخدم مجموعة بيانات لغوية مقدمة من موقع\n",
        "[manythings.org](http://www.manythings.org/anki/).\n",
        " تحتوي هذه المجموعة على أزواج لغوية في الشكل التالي:\n",
        "\n",
        "</div>\n",
        "\n",
        "```\n",
        "May I borrow this book?\t¿Puedo tomar prestado este libro?\n",
        "```\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "هناك مجموعة متنوعة من اللغات المتاحة ، ولكننا سنستخدم مجموعة البيانات الإنجليزية الإسبانية. لتبسيط الأمر ، اضفنا نسخة من مجموعة البيانات هذه على Google Cloud ، ولكن يمكنك أيضًا تنزيل نسختك الخاصة. بعد تنزيل مجموعة البيانات ، إليك الخطوات التي سنتخذها لإعدادها:\n",
        "\n",
        "1. أضف رمز *start* و *end* لكل جملة.\n",
        "2. نظف الجمل بإزالة [الأحرف الخاصة](https://owasp.org/www-community/password-special-characters).\n",
        "3.كوِّن فهرس الكلمات وفهرس الكلمات العكسي (قواميس تعطي لكل كلمة معرف وحيد والعكس بالعكس).\n",
        "4. كمل كل جملة إلى الطول الأقصى.\n",
        "\n",
        "\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kRVATYOgJs1b"
      },
      "outputs": [],
      "source": [
        "# Download the file\n",
        "path_to_zip = tf.keras.utils.get_file(\n",
        "    'spa-eng.zip', origin='http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip',\n",
        "    extract=True)\n",
        "\n",
        "path_to_file = os.path.dirname(path_to_zip)+\"/spa-eng/spa.txt\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rd0jw-eC3jEh"
      },
      "outputs": [],
      "source": [
        "# Converts the unicode file to ascii\n",
        "def unicode_to_ascii(s):\n",
        "  return ''.join(c for c in unicodedata.normalize('NFD', s)\n",
        "      if unicodedata.category(c) != 'Mn')\n",
        "\n",
        "\n",
        "def preprocess_sentence(w):\n",
        "  w = unicode_to_ascii(w.lower().strip())\n",
        "\n",
        "  # creating a space between a word and the punctuation following it\n",
        "  # eg: \"he is a boy.\" => \"he is a boy .\"\n",
        "  # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation\n",
        "  w = re.sub(r\"([?.!,¿])\", r\" \\1 \", w)\n",
        "  w = re.sub(r'[\" \"]+', \" \", w)\n",
        "\n",
        "  # replacing everything with space except (a-z, A-Z, \".\", \"?\", \"!\", \",\")\n",
        "  w = re.sub(r\"[^a-zA-Z?.!,¿]+\", \" \", w)\n",
        "\n",
        "  w = w.strip()\n",
        "\n",
        "  # adding a start and an end token to the sentence\n",
        "  # so that the model know when to start and stop predicting.\n",
        "  w = '<start> ' + w + ' <end>'\n",
        "  return w"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "opI2GzOt479E"
      },
      "outputs": [],
      "source": [
        "en_sentence = u\"May I borrow this book?\"\n",
        "sp_sentence = u\"¿Puedo tomar prestado este libro?\"\n",
        "print(preprocess_sentence(en_sentence))\n",
        "print(preprocess_sentence(sp_sentence).encode('utf-8'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OHn4Dct23jEm"
      },
      "outputs": [],
      "source": [
        "# 1. Remove the accents\n",
        "# 2. Clean the sentences\n",
        "# 3. Return word pairs in the format: [ENGLISH, SPANISH]\n",
        "def create_dataset(path, num_examples):\n",
        "  lines = io.open(path, encoding='UTF-8').read().strip().split('\\n')\n",
        "\n",
        "  word_pairs = [[preprocess_sentence(w) for w in l.split('\\t')]  for l in lines[:num_examples]]\n",
        "\n",
        "  return zip(*word_pairs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cTbSbBz55QtF"
      },
      "outputs": [],
      "source": [
        "en, sp = create_dataset(path_to_file, None)\n",
        "print(en[-1])\n",
        "print(sp[-1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OmMZQpdO60dt"
      },
      "outputs": [],
      "source": [
        "def max_length(tensor):\n",
        "  return max(len(t) for t in tensor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bIOn8RCNDJXG"
      },
      "outputs": [],
      "source": [
        "def tokenize(lang):\n",
        "  lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(\n",
        "      filters='')\n",
        "  lang_tokenizer.fit_on_texts(lang)\n",
        "\n",
        "  tensor = lang_tokenizer.texts_to_sequences(lang)\n",
        "\n",
        "  tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor,\n",
        "                                                         padding='post')\n",
        "\n",
        "  return tensor, lang_tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eAY9k49G3jE_"
      },
      "outputs": [],
      "source": [
        "def load_dataset(path, num_examples=None):\n",
        "  # creating cleaned input, output pairs\n",
        "  targ_lang, inp_lang = create_dataset(path, num_examples)\n",
        "\n",
        "  input_tensor, inp_lang_tokenizer = tokenize(inp_lang)\n",
        "  target_tensor, targ_lang_tokenizer = tokenize(targ_lang)\n",
        "\n",
        "  return input_tensor, target_tensor, inp_lang_tokenizer, targ_lang_tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GOi42V79Ydlr"
      },
      "source": [
        "### <div dir=\"rtl\"> تحديد حجم مجموعة البيانات للتجربة بشكل أسرع (اختياري)</div>\n",
        "\n",
        "<div dir=\"rtl\"> \n",
        "\n",
        "سيستغرق التدريب على مجموعة البيانات الكاملة (أكثر من 100،000 جملة) وقتًا طويلاً. للتدريب بشكل أسرع ، يمكننا تحديد حجم مجموعة البيانات إلى 30،000 جملة (بالطبع ، تنقص جودة الترجمة ببيانات أقل):\n",
        "\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cnxC7q-j3jFD"
      },
      "outputs": [],
      "source": [
        "# Try experimenting with the size of that dataset\n",
        "num_examples = 30000\n",
        "input_tensor, target_tensor, inp_lang, targ_lang = load_dataset(path_to_file, num_examples)\n",
        "\n",
        "# Calculate max_length of the target tensors\n",
        "max_length_targ, max_length_inp = max_length(target_tensor), max_length(input_tensor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4QILQkOs3jFG"
      },
      "outputs": [],
      "source": [
        "# Creating training and validation sets using an 80-20 split\n",
        "input_tensor_train, input_tensor_val, target_tensor_train, target_tensor_val = train_test_split(input_tensor, target_tensor, test_size=0.2)\n",
        "\n",
        "# Show length\n",
        "print(len(input_tensor_train), len(target_tensor_train), len(input_tensor_val), len(target_tensor_val))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lJPmLZGMeD5q"
      },
      "outputs": [],
      "source": [
        "def convert(lang, tensor):\n",
        "  for t in tensor:\n",
        "    if t!=0:\n",
        "      print (\"%d ----> %s\" % (t, lang.index_word[t]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VXukARTDd7MT"
      },
      "outputs": [],
      "source": [
        "print (\"Input Language; index to word mapping\")\n",
        "convert(inp_lang, input_tensor_train[0])\n",
        "print ()\n",
        "print (\"Target Language; index to word mapping\")\n",
        "convert(targ_lang, target_tensor_train[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgCLkfv5uO3d"
      },
      "source": [
        "### <div dir=\"rtl\"> أنشئ مجموعة بيانات من نوع tf.data </div> "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TqHsArVZ3jFS"
      },
      "outputs": [],
      "source": [
        "BUFFER_SIZE = len(input_tensor_train)\n",
        "BATCH_SIZE = 64\n",
        "steps_per_epoch = len(input_tensor_train)//BATCH_SIZE\n",
        "embedding_dim = 256\n",
        "units = 1024\n",
        "vocab_inp_size = len(inp_lang.word_index)+1\n",
        "vocab_tar_size = len(targ_lang.word_index)+1\n",
        "\n",
        "dataset = tf.data.Dataset.from_tensor_slices((input_tensor_train, target_tensor_train)).shuffle(BUFFER_SIZE)\n",
        "dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qc6-NK1GtWQt"
      },
      "outputs": [],
      "source": [
        "example_input_batch, example_target_batch = next(iter(dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNfHIF71ulLu"
      },
      "source": [
        "## <div dir=\"rtl\"> برمجة نموذج التشفير - فك التشفير (encoder - decoder) </div>\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "لنقم ببرمجة نموذج التشفير - فك التشفير ببواسطة آلية الانتباه  (attention). يمكنك أن تقرأ المزيد عن هذا النوع من النماذج في الدرس التعلمي لTensorFlow [الترجمة الآلية باستخدام الشبكات العصبية (seq2seq)](https://github.com/tensorflow/nmt).\n",
        " يستخدم هذا المثال مجموعةً أحدث من واجهات برمجة التطبيقات (APIs). \n",
        "و يستخدم هذا الدّفتر [معادلات الانتباه](https://github.com/tensorflow/nmt#background-on-the-attention-mechanism) من الدرس التعليمي seq2seq.\n",
        " يوضح الرسم البياني التالي أن كل كلمة مدخلة يتم تعيين وزن لها بواسطة آلية الانتباه التي يستخدمها نموذج فك التشفير (decoder) بعد ذلك للتنبؤ بالكلمة التالية في الجملة.\n",
        " الصورة و المعادلات أدناه هما مثال لآلية الانتباه من [ورقة بحث لونغ Luong](https://arxiv.org/abs/1508.04025v5).\n",
        "\n",
        "</div>\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg\" width=\"500\" alt=\"attention mechanism\">\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "يتم إدخال الجمل إلى نموذج التشفير الذي يعطينا ناتج التشفير بالأبعاد التّالية *(batch_size، max_length، hidden_size)* و الحالة الخفية (hidden state) لنموذج التشفير بالأبعاد التّالية *(batch_size، hidden_size)*.\n",
        "\n",
        "فيما يلي المعادلات التي يتم تطبيقها:\n",
        "\n",
        "</div>\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_0.jpg\" alt=\"attention equation 0\" width=\"800\">\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_1.jpg\" alt=\"attention equation 1\" width=\"800\">\n",
        "\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "\n",
        "يستخدم هذا الدرس التعليمي [آلية انتباه Bahdanau](https://arxiv.org/pdf/1409.0473.pdf) لنموذج التشفير. دعونا نحدد الرموز قبل كتابة النسخة المبسطة:\n",
        "\n",
        "* EO = نتاج نموذج التشفير (Encoder Ouput)\n",
        "* H = الحالة الخفية (hidden state)\n",
        "* X =  المُدخل إلى نموذج فك التشفير\n",
        "\n",
        "</div>\n",
        "\n",
        "<div dir=\"rtl\">\n",
        "    \n",
        "فيما يلي الشبه شفرة (pseudo-code)، أي الخطوات المبسّطة التي يقوم بها النموذج:\n",
        "\n",
        "* `score = FC(tanh(FC(EO) + FC(H)))`\n",
        "* `attention weights = softmax(score, axis = 1)`\n",
        "\n",
        " بدون تحديد صريح، يتم تطبيق Softmax على المحور (axis) الأخير ولكن هنا نريد تطبيقه على المحور الأول ، حيث أن شكل النتيجة هو *(batch_size، max_length، hidden_size)*. \"max_length\" هو طول مدخلاتنا. نظرًا لأننا نحاول تعيين وزن لكل مدخل ، يجب تطبيق Softmax على هذا المحور.\n",
        "* `context vector = sum(attention weights * EO, axis = 1)`. نفس السبب أعلاه \n",
        "لاختيار المحور  1. \n",
        "* `embedding output` = يتم تمرير المدخل X إلى نموذج فك التشفير  من خلال طبقة التضمين (embedding).\n",
        "* `merged vector = concat(embedding output, context vector)`\n",
        "* ثم يتم إدخال هذا المتجه (vector) المدمج لـ GRU\n",
        "\n",
        "تم تحديد أشكال جميع المتجهات (vectors) لكل خطوة في التعليقات في الكود:\n",
        "\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nZ2rI24i3jFg"
      },
      "outputs": [],
      "source": [
        "class Encoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
        "    super(Encoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.enc_units = enc_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.enc_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "\n",
        "  def call(self, x, hidden):\n",
        "    x = self.embedding(x)\n",
        "    output, state = self.gru(x, initial_state = hidden)\n",
        "    return output, state\n",
        "\n",
        "  def initialize_hidden_state(self):\n",
        "    return tf.zeros((self.batch_sz, self.enc_units))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "60gSVh05Jl6l"
      },
      "outputs": [],
      "source": [
        "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "# sample input\n",
        "sample_hidden = encoder.initialize_hidden_state()\n",
        "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
        "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
        "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "umohpBN2OM94"
      },
      "outputs": [],
      "source": [
        "class BahdanauAttention(tf.keras.layers.Layer):\n",
        "  def __init__(self, units):\n",
        "    super(BahdanauAttention, self).__init__()\n",
        "    self.W1 = tf.keras.layers.Dense(units)\n",
        "    self.W2 = tf.keras.layers.Dense(units)\n",
        "    self.V = tf.keras.layers.Dense(1)\n",
        "\n",
        "  def call(self, query, values):\n",
        "    # query hidden state shape == (batch_size, hidden size)\n",
        "    # query_with_time_axis shape == (batch_size, 1, hidden size)\n",
        "    # values shape == (batch_size, max_len, hidden size)\n",
        "    # we are doing this to broadcast addition along the time axis to calculate the score\n",
        "    query_with_time_axis = tf.expand_dims(query, 1)\n",
        "\n",
        "    # score shape == (batch_size, max_length, 1)\n",
        "    # we get 1 at the last axis because we are applying score to self.V\n",
        "    # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
        "    score = self.V(tf.nn.tanh(\n",
        "        self.W1(query_with_time_axis) + self.W2(values)))\n",
        "\n",
        "    # attention_weights shape == (batch_size, max_length, 1)\n",
        "    attention_weights = tf.nn.softmax(score, axis=1)\n",
        "\n",
        "    # context_vector shape after sum == (batch_size, hidden_size)\n",
        "    context_vector = attention_weights * values\n",
        "    context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "    return context_vector, attention_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k534zTHiDjQU"
      },
      "outputs": [],
      "source": [
        "attention_layer = BahdanauAttention(10)\n",
        "attention_result, attention_weights = attention_layer(sample_hidden, sample_output)\n",
        "\n",
        "print(\"Attention result shape: (batch size, units) {}\".format(attention_result.shape))\n",
        "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yJ_B3mhW3jFk"
      },
      "outputs": [],
      "source": [
        "class Decoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.dec_units = dec_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.dec_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "    # used for attention\n",
        "    self.attention = BahdanauAttention(self.dec_units)\n",
        "\n",
        "  def call(self, x, hidden, enc_output):\n",
        "    # enc_output shape == (batch_size, max_length, hidden_size)\n",
        "    context_vector, attention_weights = self.attention(hidden, enc_output)\n",
        "\n",
        "    # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
        "    x = self.embedding(x)\n",
        "\n",
        "    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
        "    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
        "\n",
        "    # passing the concatenated vector to the GRU\n",
        "    output, state = self.gru(x)\n",
        "\n",
        "    # output shape == (batch_size * 1, hidden_size)\n",
        "    output = tf.reshape(output, (-1, output.shape[2]))\n",
        "\n",
        "    # output shape == (batch_size, vocab)\n",
        "    x = self.fc(output)\n",
        "\n",
        "    return x, state, attention_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P5UY8wko3jFp"
      },
      "outputs": [],
      "source": [
        "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "sample_decoder_output, _, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),\n",
        "                                      sample_hidden, sample_output)\n",
        "\n",
        "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ch_71VbIRfK"
      },
      "source": [
        "## <div dir='rtl'> تحديد خوارزميّة تحسين (optimizer) ودالّة خسارة (loss function) </div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WmTHr5iV3jFr"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMVWzzsfNl4e"
      },
      "source": [
        "## <div dir='rtl'> حفظ نقاط الفحص   عبر الكائنات (objects) </div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Zj8bXQTgNwrF"
      },
      "outputs": [],
      "source": [
        "checkpoint_dir = './training_checkpoints'\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
        "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "                                 encoder=encoder,\n",
        "                                 decoder=decoder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpObfY22IddU"
      },
      "source": [
        "## <div dir='rtl'> التدريب </div>\n",
        "\n",
        "<div dir='rtl'>\n",
        "\n",
        "1. مرر *المُدخل* عبر *نموذج التشفير* الذي يقوم بإخرج نتاج نموذج التشفير و *الحالة المخفية*.\n",
        "2. يتم تمرير نتاج نموذج التشفير ، والحالة المخفية ومُدخل نموذج فك التشفير (وهو الرمز المميز *start*) إلى نموذج فك التشفير.\n",
        "3. يقوم نموذج فك التشفير بإرجاع *التنبؤات* والحالة المخفية.\n",
        "4. يتم بعد ذلك تمرير الحالة المخفية لنموذج فك التشفير إلى النموذج واستخدام التنبؤات لحساب دالة الخسارة.\n",
        "5. استخدم إجبار المعلم (Teacher forcing) لتحديد المُدخل التالي إلى نموذج فك التشفير.\n",
        "6. *إجبار المعلم* هو التقنية الذي يتم خلالها تمرير *الكلمة الهدف أي الصحيحة* كالمدخل التالي إلى نموذج فك التشفير.\n",
        "7. الخطوة الأخيرة هي حساب الميول (gradients) وتطبيقها في خوارزميّة التحسين ثم تطبيق الانتشار العكسي (backpropagation).\n",
        "\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sC9ArXSsVfqn"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_step(inp, targ, enc_hidden):\n",
        "  loss = 0\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
        "\n",
        "    dec_hidden = enc_hidden\n",
        "\n",
        "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "\n",
        "    # Teacher forcing - feeding the target as the next input\n",
        "    for t in range(1, targ.shape[1]):\n",
        "      # passing enc_output to the decoder\n",
        "      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
        "\n",
        "      loss += loss_function(targ[:, t], predictions)\n",
        "\n",
        "      # using teacher forcing\n",
        "      dec_input = tf.expand_dims(targ[:, t], 1)\n",
        "\n",
        "  batch_loss = (loss / int(targ.shape[1]))\n",
        "\n",
        "  variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return batch_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ddefjBMa3jF0"
      },
      "outputs": [],
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "\n",
        "  enc_hidden = encoder.initialize_hidden_state()\n",
        "  total_loss = 0\n",
        "\n",
        "  for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):\n",
        "    batch_loss = train_step(inp, targ, enc_hidden)\n",
        "    total_loss += batch_loss\n",
        "\n",
        "    if batch % 100 == 0:\n",
        "      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                                   batch,\n",
        "                                                   batch_loss.numpy()))\n",
        "  # saving (checkpoint) the model every 2 epochs\n",
        "  if (epoch + 1) % 2 == 0:\n",
        "    checkpoint.save(file_prefix = checkpoint_prefix)\n",
        "\n",
        "  print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                      total_loss / steps_per_epoch))\n",
        "  print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mU3Ce8M6I3rz"
      },
      "source": [
        "## <div dir='rtl'> الترجمة </div>\n",
        "\n",
        "<div dir='rtl'>\n",
        "\n",
        "* تشبه خطوات وظيفة التقييم التّالية خطوات عمليّة التدريب ، باستثناء أننا لا نستخدم تقنية *إجبار المعلم* هنا. المُدخل في نموذج فك التشفير في كل خطوة زمنية هو تنبؤاته السابقة بالإضافة إلى الحالة المخفية و نتاج نموذج التشفير.\n",
        "* أوقف التنبؤ عندما يتنبأ النموذج برمز *end*.\n",
        "* احفظ *أوزان الانتباه* لكل خطوة زمنية.\n",
        "\n",
        "Note: \n",
        "يتم احتساب نتاج نموذج التشفير مرة واحدة فقط لمُدخل واحد.\n",
        "\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EbQpyYs13jF_"
      },
      "outputs": [],
      "source": [
        "def evaluate(sentence):\n",
        "  attention_plot = np.zeros((max_length_targ, max_length_inp))\n",
        "\n",
        "  sentence = preprocess_sentence(sentence)\n",
        "\n",
        "  inputs = [inp_lang.word_index[i] for i in sentence.split(' ')]\n",
        "  inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
        "                                                         maxlen=max_length_inp,\n",
        "                                                         padding='post')\n",
        "  inputs = tf.convert_to_tensor(inputs)\n",
        "\n",
        "  result = ''\n",
        "\n",
        "  hidden = [tf.zeros((1, units))]\n",
        "  enc_out, enc_hidden = encoder(inputs, hidden)\n",
        "\n",
        "  dec_hidden = enc_hidden\n",
        "  dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
        "\n",
        "  for t in range(max_length_targ):\n",
        "    predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
        "                                                         dec_hidden,\n",
        "                                                         enc_out)\n",
        "\n",
        "    # storing the attention weights to plot later on\n",
        "    attention_weights = tf.reshape(attention_weights, (-1, ))\n",
        "    attention_plot[t] = attention_weights.numpy()\n",
        "\n",
        "    predicted_id = tf.argmax(predictions[0]).numpy()\n",
        "\n",
        "    result += targ_lang.index_word[predicted_id] + ' '\n",
        "\n",
        "    if targ_lang.index_word[predicted_id] == '<end>':\n",
        "      return result, sentence, attention_plot\n",
        "\n",
        "    # the predicted ID is fed back into the model\n",
        "    dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "  return result, sentence, attention_plot"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s5hQWlbN3jGF"
      },
      "outputs": [],
      "source": [
        "# function for plotting the attention weights\n",
        "def plot_attention(attention, sentence, predicted_sentence):\n",
        "  fig = plt.figure(figsize=(10,10))\n",
        "  ax = fig.add_subplot(1, 1, 1)\n",
        "  ax.matshow(attention, cmap='viridis')\n",
        "\n",
        "  fontdict = {'fontsize': 14}\n",
        "\n",
        "  ax.set_xticklabels([''] + sentence, fontdict=fontdict, rotation=90)\n",
        "  ax.set_yticklabels([''] + predicted_sentence, fontdict=fontdict)\n",
        "\n",
        "  ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "  ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sl9zUHzg3jGI"
      },
      "outputs": [],
      "source": [
        "def translate(sentence):\n",
        "  result, sentence, attention_plot = evaluate(sentence)\n",
        "\n",
        "  print('Input: %s' % (sentence))\n",
        "  print('Predicted translation: {}'.format(result))\n",
        "\n",
        "  attention_plot = attention_plot[:len(result.split(' ')), :len(sentence.split(' '))]\n",
        "  plot_attention(attention_plot, sentence.split(' '), result.split(' '))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n250XbnjOaqP"
      },
      "source": [
        "## <div dir='rtl'>استعادة أحدث نقطة فحص واختبارها</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UJpT9D5_OgP6"
      },
      "outputs": [],
      "source": [
        "# restoring the latest checkpoint in checkpoint_dir\n",
        "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WrAM0FDomq3E"
      },
      "outputs": [],
      "source": [
        "translate(u'hace mucho frio aqui.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zSx2iM36EZQZ"
      },
      "outputs": [],
      "source": [
        "translate(u'esta es mi vida.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A3LLCx3ZE0Ls"
      },
      "outputs": [],
      "source": [
        "translate(u'¿todavia estan en casa?')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DUQVLVqUE1YW"
      },
      "outputs": [],
      "source": [
        "# wrong translation\n",
        "translate(u'trata de averiguarlo.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTe5P5ioMJwN"
      },
      "source": [
        "## <div dir='rtl'> الخطوات التالية </div>\n",
        "\n",
        "<div dir='rtl'>\n",
        "\n",
        "* [قم بتنزيل مجموعة بيانات مختلفة](http://www.manythings.org/anki/) لتجربة ترجمات أخرى، على سبيل المثال ، الإنجليزية إلى الألمانية ، أو الإنجليزية إلى الفرنسية.\n",
        "* جرب التدريب على مجموعة بيانات أكبر ، أو استخدم المزيد من الحقبات (epochs).\n",
        "\n",
        "</div>\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "nmt_with_attention.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
